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MOTIVATION 

The  purpose  of  this  paper  is  to  explore  non-traditional  visualization  techniques  to  assess  threat  and  other 
variables  in  command  center  settings.  Not  all  entities  that  a  commander  interacts  in  real  life  can  be 
converted  to  visual  medium,  which  is  but  just  one  of  the  five  senses  that  we  have.  As  an  example,  sense  of 
threat  during  a  critical  situation  during  a  mission  cannot  be  easily  communicated  with  color.  Many  special 
skills  such  as  a  surgeon  using  a  sharp  scalpel  or  a  Pilot  negotiating  a  sharp  curve  cannot  be  easily 
represented  in  videos  or  even  real  time  Virtual  Environments.  We  call  such  systems  -  non-physical 
visualization.  While  elsewhere  in  this  conference  we  speak  about  Ontological  representation  of  such 
events,  here  some  techniques  currently  underdevelopment  to  ‘visualize’  such  parameters  are  presented  - 
including  tactile  and  haptic  rendering  techniques. 


BACKGROUND 

Usually  visualizing  battlefield  implies  maps,  computer  screens  filled  with  information  and  perhaps  3-D 
displays  that  represent  a  wide  range  of  data  ranging  from  intel,  logistics  and  threat  information.  Decision 
of  a  commander  is  often  based  on  understanding  a  threat  level  and  calculating  a  probability  of  success 
given  all  the  various  sets  of  information.  Some  problems  related  to  decision  making  process  are: 

•  Too  much  data,  too  little  knowledge 

•  Complex  reasoning  that  cannot  be  represented  graphically 

•  Problems  in  drawing  attention  to  some  elements  due  to  information  overload 

•  Trust  and  confidence  in  the  visual  data  presented 

How  adding  larger,  more  complex  visualization  system  can  solve  much  of  this  is  a  matter  of  debate. 
Is  there  a  more  sophisticated  way  to  address  some  of  the  problems?  In  the  recent  years  many  interesting 
new  concepts  have  been  developed  in  the  area  Physically  based  modeling  which  could  be  used  to  tackle 
some  of  the  problems.  One  such  idea  is  to  use  haptics  rendering  and  modeling  to  represent  information  in 
a  new  dimension. 

Haptics,  from  the  Greek  word  Haptikos  is  essentially  the  sense  of  touch.  More  broadly  in  our  context  it 
can  be  referred  to  as  a  resistive  or  feedback  force.  With  the  advent  of  haptics  and  the  ability  to  add  the 
sense  of  touch  to  a  virtual  environment,  it  is  now  possible  to  develop  completely  new  set  of  interfaces  for 
decision  support.  Information,  such  as  threat,  risk,  confidence  levels  can  be  converted  to  variety  of 
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physical  sensations  that  have  no  equivalent  representation  in  the  graphics  or  visual  environments. 
There  are  many  forms  of  such  force  rendering  such  as;  vibration,  resistance,  viscosity  etc. 


NEW  VISUALIZATION  INTERFACE 

A  new  command  support  interface  is  proposed  here  that  combine  sense  of  touch  with  a  3-D  interface  as 
shown  in  the  figure  1.  A  3-D  interface  is  created  with  one  virtual  wall  representing  a  map  or  terrain  of  the 
battlespace  or  the  field  where  military  is  involved.  On  a  perpendicular  virtual  wall  a  haptic  rendering  of 
the  space  is  created.  When  a  certain  region  is  clicked  on  the  map,  a  blown  up  representation  is  presented  in 
haptic  wall  shown  as  color  grids  in  the  figure.  In  the  haptic  wall  information  such  as  threat  is  represented 
mathematically  and  converted  into  vibration  or  viscosity. 


Figure  1:  Interface  for  Haptics  in  a  Control  Room. 


MODES  OF  INTERACTION 

One  example  of  such  a  rendering  is  the  probability  of  the  existence  of  a  mine  or  a  cave,  which  is  not 
obvious  from  the  map  or  intelligence  data.  During  a  decision  support  activity,  the  commander  uses  a 
Phantom™  type  interaction  device  (figure  2)  instead  of  a  mouse  to  navigate  around  the  battlefield. 


Figure  2:  Phantom™  -  A  Commercially  Available  Haptics  Device  from  Sensable  Technologies. 

The  haptic  rendering  automatically  controls  and  keeps  track  of  the  commander’s  actions  in  the  given  space 
and  dynamically  reduces  or  increases  viscosity.  Thus  when  visual  map  calls  for  a  move  in  certain 
direction,  existence  of  high  risk  will  increase  the  resistance  of  the  navigation  task  thus  reminding  the 
commander  about  the  danger  involved  in  the  present  decision.  By  using  this  system,  visual  dimension  can 
be  freed  up  to  represent  other  important  information. 
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Results  of  early  research  have  shown  that  haptically  represented  probability  of  threat  while  moving 
towards  the  goal  is  a  viable  control  strategy.  In  a  preliminary  human  subject  test  it  was  noticed  that  users 
trust  and  adapt  to  haptics  as  to  visual  feedback  system  easily. 


DISCUSSION 

New  non-traditional  visual  interface  obviously  opens  up  a  number  of  open-ended  questions.  We  are  posing 
a  few  here  for  discussion: 

•  Will  commanders  trust  haptics  or  a  combination  of  visual  and  haptics  system? 

•  What  information  is  suitable  for  non-traditional  visuals? 

•  What  is  the  range  of  such  devices? 

•  What  is  the  optimum  combination  of  3-D  VR  technology  and  Haptics? 


RTO-MP-105 
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SYMPOSIA  DISCUSSION  -  PAPER  NO:  8 

Author  Name:  Dr.  T.  Kesavadas,  State  University  at  Buffalo,  USA 

Comment: 

There  is  the  issue  of  having  to  know  what  questions  to  ask  to  drill  down  to  more  info.  There  is  the  trade 
off  between  providing  enough  information  to  the  user  without  clutter. 

Comment: 

Response  from  users  is  positive,  but  there  is  a  need  to  train  people  to  use  the  techniques. 
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Non- traditional  Visualization 


Profs.  T.  Kesavadas  and  J  ames  Llinas 
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Problems  in  Large  Scale  Viz. 


♦  Not  every  information  fusion  parameters  can 
be  presented  visually  -  Non  physical 
visualization 

■  Fear 

■  Uncertainty 


♦3-D  data  visualization  can  overload  human 
cognition 


♦  Visualizing  certain  data  in  visual  form  is  non 
intuitive 
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Visualization,  I  nfo.  Fusion  and 
Logical  Constructs 


Transforms  -  Modalities  Mental  Models  Ontology 


Physical _ Non  physical 

Traditional  Viz.  - *■  Non  Traditional 


Audio  Haptics 


2-D  Screens 


3-D  stereo  glasses  HMD  CAVE 


8-3 


Presentation  Agenda 


Phases 


Introduction 


ocus  &  goals  of 
Research 


Review 


-  Non  traditional 


Potential  of  Non 

traditional  Viz.  ~  Haptics 


Problems  in 
Visualization 


University  at  Buffalo 


State  University  of  < 


Our  Work  Conclusion 


-Two  experiments 


8-4 


University  at  Buffalo 

Stott  UmmtytfNw  York 


Sensory  perceptions 


Taste 


Smell 


Some  Examples 
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Illuminating  Clay 


•Landscape  models  are  constructed  using  a  ductile  clay  support. 


•Three-dimensional  geometry  is  captured  in  real  time  using  a  laser 
scanner. 


•From  this  information  simulations  such  as  shadow  casting,  land 
erosion,  visibility  and  traveling  time  are  calculated. 

•Finally,  the  results  are  projected  back  onto  the  clay  model. 


MIT  Tangible  Media  Lab 
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Ambient  Room 
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The  ambient  ROOM  allows  users  to  be 
aware  of  background  bits  using  ambient 
display  media  such  as  ambient  light,  shadow, 
sound,  airflow,  water  movement  in  an 
augmented  architectural  space. 


phctfo  flrygg  JJVmar 


Ghostly  Presence  is  an  application  of  the 
ambientROOM  platform.  Ghostly  Presence  explores 
the  subtle  representation  of  the  "presence"  of  people 
mediated  by  the  physical  environment. 
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SandScape 


SandScape  is  a  tangible  interface  for 
designing  and  understanding 
landscapes  through  a  variety  of 
computational  simulations  using  sand. 
Users  view  these  simulations  as  they 
are  projected  on  the  surface  of  a  sand 
model  that  represents  the  terrain. 


The  users  can  alter  the  form  of  the  landscape  model  by 
manipulating  sand  while  seeing  the  resultant  effects  of 
computational  analysis  generated  and  projected  on  the  surface  of 
sand  in  real-time. 
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Actuated  Workbench 


The  Actuated  Workbench  is  a  device  that  uses  magnetic 
forces  to  move  objects  on  a  table  in  two  dimensions.  It  is 
intended  for  use  with  existing  tabletop  tangible  interfaces, 
providing  an  additional  feedback  loop  for  computer  output, 
and  helping  to  resolve  inconsistencies  that  otherwise 
arise  from  the  computer's  inability  to  move  objects  on  the 
table. 
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Ping  Pong  Plus 


TBl 


Movie 


Photos.  Chappell 

PingPongPlus  is  a  digitally  enhanced  version  of  the 
classic  ping-pong  game.  It  is  played  with  ordinary,  un¬ 
tethered  paddles  and  balls,  and  features  a  "reactive  table" 
that  incorporates  sensing,  sound,  and  projection 
technologies.  Projectors  display  patterns  of  light  and 
shadow  on  the  table;  bouncing  balls  leave  images  of 
rippling  water;  and  the  rhythm  of  play  drives  accompanying 
music  and  visuals.  8-n 
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The  BreakBits  interface  is  intended  as  a 
complement  to  the  traditional,  mainly 
graphical,  interface.  The  aim  is  to  help 
the  user  concentrate  on  the  most  relevant 
information  by  giving  musical  guidance. 
Rather  than  directly  mapping  different 
sounds  to  specific  events,  the  purpose  is 
to  continuously  present  information  about 
the  status  of  the  computing  environment 
processes  in  an  emotional  and  discreet 
manner. 

Maintaining  focus  may  be  hard  or  physically  impossible  (e.g.  when 
turning  away  from  the  interface),  and  the  amount  of  information  may  be 
overwhelming.  In  such  a  case,  the  solution  to  the  problem  needs  to  be 
non-visual,  ideally  an  auditory  interface. 

BreakBits  Prototype:  http://www.viktoria.se/plav/proiects/breakbits/eval.html 


AUDIO  BreakBits 


m 
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Our  approach 


♦  Study  other  sensory  modes  of  data 
presentation 

■  Haptics 

■  Tactile 
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Virtual  Lexicon 


Haptic  feedback 

The  sensation  of  weight  or  resistance  in  a  virtual 
world. 


(from  the  Greek  haptesthai,  meaning  to  touch)- 
Cyber  Edge  Journal, [1993b] 


Tactile  Feedback 


Sensation  applied  to  the  skin,  typically  in  response 
to  contact  or  other  actions  in  a  virtual  world. 


Examples  of  Touch 

Vibration,  Viscosity,  Resistance,  Constraint,  Pain, 
Temperature 
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Sensable  Technologies 


Desktop 


Premium  1.0 
with  encoder 
gimbal 


Premium  1.5  with 
finger  stylus 


Reachln  Interfaces 


Potential  of  Haptics 


Analysis 
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Awareness 


uman  computei 
interaction 
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Application  to  design  space  optimization 
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•  One  of  the  ways  in  which  a  design  problem  is  solved  is  by 
breaking  a  design  process  into  a  series  of  stages. 

•  Designer  may  make  decisions  at  one  stage  before  moving  on 
to  subsequent  stages. 

•  Making  a  decision  may  simplify  the  search  problem  by 
limiting  search  to  a  subset  of  the  overall  state  space 

•  Typical  of  this  class  of  optimization  techniques  are  the 
pattern  search  algorithm  by  Hooke  and  Jeeves  (1961), 
sequential  search  by  Spendley,  Hextand  Himsworth  (1962). 
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Computational  algorithm  and  methodology  for  haptics 
based  solution 


Step  1:  Load  design  space  optimization  scene  graph, 
with  coordinate  axes,  and  constraint  contours,  with 
convergence  tolerance  equal  to  the  radius  of  the  central 
sphere. 

Step  2:  Start  with  an  initial  vector  X,  and  compute 
function  value  at  the  point  fr 


Step  3:  Perform  a  cycle  of  random  moves,  by  moving 
the  phantom  stylus  in  the  design  space  each  along  one 
coordinate  direction. 

Step  4:  Compute  X+  =  X,  +  AX,  fj  =  f  (XT),  Af  =  f*  -  fi 


Design  space 

•  This  coordinate  information 
of  the  probe  is  used  to 
represent  design  variables. 


Step:  5:  Return  a  force  given  by  the  force  relation 

Fi=-(k*m:mxi\)*xi 

Step  6:  Use  the  phantom  stylus  to  guide  the  designer  in 
the  direction  of  force  effects  subject  to  constraints  and 
axes  specified. 


•  The  user  can  perform  an 
exploratory  search  of  the 
region  to  arrive  at  the  optimal 
solution. 


Step  1:  Test  for  convergence. 
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B0B 


/  Figure  No.  1 


BEE 


£ile  Edit  lools  Window  Help 

I'D  &  H  S  \  A  ?  /  MO 


Programming  Environment:  MATLAB  and  GHOST  on  Windows  NT 

•The  user  uses  the  phantom  probe  to  move  blocks  into  cells. 

•  A  cost  value  is  associated  with  each  configuration  of  blocks. 


•  Information  is  represented  through  interactive  plots  and/or 
force  effects. 


•  One  set  of  experiments  is  complete  when  an  optimal  cost  is 
reached. 
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Inferences 


discernable. 

•The  combined  graphic  and  haptics  approach  was  preferred 
by  80%  of  the  users. 

•  The  haptics  approach  takes  lesser  time  in  66.67  %  of 
cases. 

•  Haptics  provides  an  efficient  duplex  sense  for  decision 
making. 
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Uncertainty  Display 

1 Experiment  1 1 1 1 1 1 1 

With  Profs.  Ann  Bizantz  and  J  ames  Llinas 
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Experimental  Design 


♦  Display  of  probability:  Visual,  Tactile,  or 
Auditory 

♦  “Look”  before  move  -  make  task  consistent 
across  modalities 

♦  Tradeoff  between  risky  path  and  shortest  path 

♦  Score  motivation:  length,  time  vs.  mines  “hit”; 
monetary  bonus  for  high  score 

♦  Measure  path  length,  speed,  risk,  trust  in 
display 
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Haptics  for  Threat  Perception 

'"i  f"  Terrain  visualization  with  distribution 

corresponding  to  mine  location 

Color  code  corresponds  to  probability  of  mine  field 
location  between  0  and  1  in  steps  of  0.1 

The  user  makes  a  decision  depending  upon  a 
combination  of  different  Effects  namely: 

•  Visual 

•  Audio 

•  Haptic 
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Experimental  Setup 

|  start] 

-inlx! 


O  Instructions 

List  of  Instructions 


1 .  To  Start :  Right  Click  on  the  Start  Button 
Z.  Press  Left  Button  to  try  out  a  square 
3.  Press  Right  Button  to  confirm  a  square 

Remember  to  hold  the  haptic  device  well 


Start  I  J  i^Jj  ;  j  I^IVisualExps  |  SO  main  -  Microsoft..., |  1^  Adobe  Photoshop  |  "D:\users\5ant...  jffl  Experimental ...  SSSli 


|  6:17  PM 
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Experimental  Design 


PI 

P2 

P3 


20  trials  20  trials  20  trials 
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Performance  Results 


Auditory  Haptic  Visual  Auditory  Haptic  Visual 


Modality 


Modality 
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Summary 


♦  Visual  Condition  resulted  in  more  conservative 
performance  -  less  risk,  but  longer  path  length. 
Auditory  condition  had  shortest  path  length,  but 
more  risk. 

♦  Visual  Condition  initially  faster,  but  with  learning, 
haptic  condition  approached  these  times 

♦  Auditory  remained  slower 

♦  Overall  workload  similar,  but  haptic  had  higher 
physical  workload,  and  auditory  condition  had  higher 
levels  of  frustration 

♦  Trust  similar  across  modalities,  but  tended  to 
decrease  with  time 
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Observation 


♦  For  risk  avoidance,  visual  condition  preferable 

♦  For  condition  where  visual  clutter  is 
problematic,  haptic  interface  provided 
acceptable  performance  in  terms  of  risk 
avoidance 

♦  Frustration  level  for  haptic  interface  less  than 
auditory 
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How  haptic  maps  can  be 
integrated  with  Visual  System 
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Various  sensory  media 


♦  Vibration 

♦  Holography 

♦  Resistance 

♦  Autostereoscopy 

♦  Viscosity 

♦  Augumented  Reality 

♦  Temperature 

♦  Merged  Visual +GUI 

♦  Wind 

♦  Fishpond  VR 

♦  Ambient  Light 

♦  6- Wall  CAVES 

♦  Focused  Light 
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Topics  for  discussion 


♦  What  is  the  ideal  interaction  medium? 

♦  Matching  mental  model? 

♦  What  types  of  other  information 
visualization  can  be  provided? 

♦  Ontology? 
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I  nformation  and  modes  of  presentation 


Vibration 

Viscosity 

Resistance 

Uncertainly 

Constraints 

Boundaries 

Geography 
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□  y 

■  n 

□  Both/Cannot  decide 


0 


EL 


LL 


Is  there 
Ease  of 
moving 
blocks 
around? 


Can 

difference 
between 
levels  of 
viscous 
effects  be 
perceived? 


Which  of  the 
tasks,  is 
preferable 
haptic  /non- 
haptic  or  a 
combination 
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